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Abstract. Recently, John Franks and Michael Handel proved 
that, for g > 3 and n < 2g— 4, every homomorphism from the map- 
ping class group of an orientable surface of genus g to GL(n, C) is 
trivial. We extend this result to n < 2g — 1, also covering the case 
.9 = 2. As an application, we prove the corresponding result for 
nonorientable surfaces. Another application is on the triviality of 
homomorphisms from the mapping class group of a closed surface 
of genus g to Aut(F„) or to Out(i^„) for n < 2g — 1. 



1. Introduction 

Let S" be a compact connected oriented surface of genus g > I with 
g > boundary components and with p > marked points in the 
interior. Let Mod(S') denote the mapping class group of S, the group 
of isotopy classes of orientation-preserving self-diffeomorphisms of S. 
Diffeomorphisms and isotopies are assumed to be the identity on the 
marked points and on the boundary. 

There is the classical representation of Mod(S') onto the symplectic 
group Sp(2(yf, Z) induced by the action of the mapping class group on 
the first homology of the closed surface of genus g. In [6], Question 1.2, 
Franks and Handel asked whether every homomorphism Mod(S') — > 
GL(n, C) is trivial for n < 2g — 1. They proved that, in fact, this is 
the case for g > 3 and n < 2g — 4, improving a result of Funar [7] 
who showed that every homomorphism from the mapping class group 
to SL(n, C) has finite image for n < a/^~+T. 

The aim of this paper is to give a complete answer to the question of 
Franks and Handel, and then consider the corresponding problem for 
nonorientable surfaces. We first prove the following theorem. 

Theorem 1. Let g > I and let n < 2g-l. Letcj) : Mod(S') GL(r2,C) 
be a homomorphism. Then the image lm(0) of is 

(1) trivial if g > 3, 

(2) a quotient of the cyclic group Zio of order 10 if g = 2, and 
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(3) a quotient 0/Z12 if{g,q) = (1,0) and ojTH^ if g = 1 and q > 1. 

I was informed by Bridson ^ that he conjectured the above theorem. 
He hints this in the page 2 of [2]. 

The first corollary to Theorem [1] is the following. 

Corollary 2. Let g >2,n<2g — l,Thea quotient o/Mod(5') and let 
: r — 7- GL(n, C) be a homomorphism. Then \m{ip) is trivial if g > 3, 
and is isomorphic to a quotient o/Zio if g = 2. 

Note that the groups Sp(2g, Z), Sp{2g, Z^), PSp(25f, Z), and PSp{2g, Z.^) 
are quotients of Mod(S'). Since Mod(S') is residually finite i9|lllj, there 
are many finite quotients. 

In the definition of the mapping class group, if we allow the diffeo- 
morphisms of S to permute the marked points, then we get a group 
A^(S'), which contains Mod(S') as a subgroup of index p\. 

Corollary 3. Let g > 2 and letn<2g-l. Let (p : M{S) GL{n, C) 
be a homomorphism. Then lm{(f)) is finite. 

For a nonorientable surface N of genus g with p > marked points, 
we define the mapping class group Mod(iV) of to be the group of 
isotopy classes of diffeomorphisms preserving the set of marked points 
(isotopies are assumed to fix the marked points). The action of a map- 
ping class on the first homology of the closed surface obtained by 
forgetting the marked points give rise to an automorphism of Hi{N] Z) 
preserving the associated Z2-valued intersection form. It was proved by 
McCarthy and Pinkall [19j, and also by Gadgil and Pancholi [8j, that, in 
fact, all automorphisms of Hi{N; Z) preserving the Z2-valued intersec- 
tion form are induced by diffeomorphisms. By dividing out the torsion 
subgroup of Hi{N; Z), we get a representation Mod(A^) — )■ GL{g—l, C). 
It is now natural to ask the triviality of the lower dimensional repre- 
sentations of Mod(iV). Since the mapping class group Mod(A^) has 
nontrivial first homology, we cannot expect that every such homomor- 
phism is trivial. Instead, one may ask the following question. 

Question 1.1. Let g > 3, and let n < g — 2. Is the image of every 
homomorphism cf) : Mod(A^) — t- GL(r7,,C) finite? 

As an application of Theorem [1], we answer this question leaving only 
one case open; the case g is even and n = g — 2. 

Theorem 4. Let g > 3, and let n < g — 2 if g is odd and n < g — 3 
if g is even. Let N be a nonorientable surface of genus g with p > 
marked points and let : Mod(A^) — )■ GL(n, C) be a homomorphism. 
Then lm(0) is finite. 
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The mapping class group of a nonorientable surface with boundary 
components may also be considered, but we restrict ourself to surfaces 
with marked points only in order to make the proof simpler. 

As another application of Theorem [H we prove the following result 
on the homomorphisms from the mapping class group of a closed ori- 
entable surface to Aut(F„) and to Out(F„), where F„ is the free group 
of rank n. Compare Theorem [5] with Question 16 in 

Theorem 5. Let g > 2 and let S be a closed orientable surface of genus 
g. Let n he a positive integer with n < 2g — 1. Let H denote either 
of Ant (Fn) or Out(F„) and let (p : Mod(S') H be a homomorphism. 
Then the image of ip is 

(1) trivial if g >3, and 

(2) a quotient o/Zio if g = 2. 

In [6], the main theorem, Theorem 1.1, is proved by induction on g. 
It is first proved for the cases g > 3 and n < 2. The main improvement 
of this paper is that we can start the induction from the cases g = 2 and 
77. < 3. The rest of the proof of Theorem [T] follows from the arguments 
of [B]. We give a slight modification of this proof. The proofs in cases 
g = 2 and n < 2 follow from the proof of Lemma 3.1 of [0] with the 
additional information that the commutator subgroup of Mod(S') is 
perfect. In the case {g,n) = (2,3) we need to treat all possible Jordan 
forms of the image of the Dehn twist about a nonseparating simple 
closed curve. 

Acknowledgments. This paper was written while I was visiting the 
Max-Planck Institut fiir Mathematik in Bonn. I thank MPIM for its 
generous support and wonderful research environment. After the com- 
pletion of the first version of this work, I was informed by John Franks 
and Michael Handel that they also improved Theorem 1.1 in [6J to the 
cases g > 2 and n < 2g — 1. I would like to thank them for sending 
the new version of their paper, which now appears on Arxiv as version 
3. I also thank Martin Bridson for his interest in this work. 

2. Algebraic preliminaries 

We state two properties of subgroups of GL(n, C). They are either 
well-known, or easy to prove. Therefore, we do not prove them. These 
properties will be used in the proof of Theorem [H 

I w * * 
z * \ and D = \ w * 
z J \ w 

elements o/GL(3,C). Then CDC = DCD if and only if C 
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Lemma 2.2. The subgroup o/GL(n, C) consisting of upper triangular 
matrices is solvable. 

We will also require the following lemma from [6]. 

Lemma 2.3. ([B], Lemma 2.2) Let G be a perfect group, and H a 
solvable group. Then any homomorphism G ^ H is trivial. 

3. Mapping class groups and commutator subgroups 

Let 5 be a compact oriented surface of genus g with p > marked 
points and with g > boundary components. In this section we give 
the results on mapping class groups required in the proof of Theorem [TJ 
For further information on mapping class groups, the reader is referred 
to [12], or [5]. For a simple closed curve a on S* we denote by ta the 
(isotopy class of the) right Dehn twist about a. 

Theorem 3.1. ([ITj, Theorem 1.2) Let g > 1. Suppose that a and 
b are two nonseparating simple closed curves on S. Then there is a 
sequence 

a = flO) o,2i ■ ■ ■ , ctk = b 
of nonseparating simple closed curves such that aj_i intersects at 
only one point 

Theorem 3.2. Let g > 2. Then the mapping class group Mod(S') is 
generated Dehn twists about nonseparating simple closed curves on S . 

Theorem 3.3. ([17j, Theorem 2.7) Let g > 2. Let a andb be two non- 
separating simple closed curves on S intersecting at one point. Then 
the commutator subgroup o/Mod(S') is generated normally by tat~i^^ . 

Theorem 13.31 should be interpreted as follows: If a normal subgroup 
of Mod(S') contains tat~f^^, then it contains the commutator subgroup of 
Mod(5'), the (normal) subgroup generated by all commutators [x,y] = 
xyx~^y~^. 

Theorem 3.4. ([17J, Theorem 4.2) Let g > 2. Then the commutator 
subgroup of Mod(S') is perfect. 

Note that, in [17J, the group Mod(S') of this paper is denoted by 
VMs. In that paper. Theorems 13.31 and 13.41 above are proved for sur- 
faces with marked points (=punctures), but the same proof apply to 
surfaces with boundary as well since Mod(S') is generated by Dehn 
twists about nonseparating simple closed curves. 

We record the following well-known relations among Dehn twists. 

Lemma 3.5. Let a and b be two simple closed curves on S, and let ta 
and tf, denote the right Dehn twists about them. 



LINEAR REPRESENTATIONS OF MAPPING CLASS GROUP 



5 



(1) If a and b are disjoint, then t a and ti, eommute. 

(2) // a intersects h transversely at one point, then they satisfy the 
braid relation tatbta = tbiah- 

Recall that the first homology group Hi{G; Z) of a group G is iso- 
morphic to the abelianization G/G', where G' is the (normal) subgroup 
of G generated by all commutators [(71,(72]. 

Theorem 3.6. ([15], Theorem 5.1) Let g > 1. Then the first homology 
group Hi(Mod{S);Z) is 

(1) trivial if g >3, 

(2) isomorphic to the cyclic group of order 10 if g = 2, 

(3) isomorphic to the cyclic group of order 12 if{g,q) = (1,0), and 

(4) isomorphic to Z'^ if g = 1 and q > I. 

Note that since any two Dehn twists about nonseparating simple 
closed curves are conjugate in Mod(S'), their classes in Hi{Mod{S); Z) 
are equal. In particular, we conclude the next lemma. 

Lemma 3.7. Let g > 1, and let b and c be two nonseparating simple 
closed curves on S. If H is an abelian group and if (p '■ Mod(S') — ?■ H 
is a homomorphism, then (pith) = 4>(tc). 

4. HOMOMORPHISMS Mod(S') GL(n, C) 

In this section we prove Theorem [TJ So let n < 25? — 1 and let 
(f) : Mod(5') — 7- GL{n, C) be a homomorphism. 

For the proof, we adopt the proof of Theorem 1.1 in [B]. Franks and 
Handel use the fact that when g > 3 the group Mod(S') is perfect. If 
this is rephrased as "the commutator subgroup of Mod(5') is perfect" 
then it is still true for the case g = 2 as well, and that is what we use 
below. The proof given in [6] for the case g > 3 and n < 2, also works 
for the case g = 2 and n < 2 with a slight modification. For the case 
g = 2 and n = 3, we need to analyze six possible Jordan forms of the 
image of the Dehn twist about a nonseparating simple closed curve. 
Once Theorem [T] is established for the cases g = 2 and n < 3, we then 
again follow a modified version of the idea of Franks and Handel to 
induct g. 

Following [6j, for a simple closed curve x on S' we denote 4>(tx) by 
Lx- If A is an eigenvalue of a linear operator L, the corresponding 
eigenspace is denoted by E\{L). We write for Ex{Lx). 

Proposition 4.1. Let g = 2 and n < 2. Then lm(0) is a quotient of 
the cyclic group Ziq. 
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Proof. If n = 1 then GL(n, C) = C* is abelian. Hence, factors 
through the first homology of Mod(S'), which is isomorphic to Ziq. So 
assume that n = 2. In this proof, we set G = Mod(S'). 

Let a, b and c be three nonseparating simple closed curves on S such 
that a is disjoint from bU c, and that b intersects c transversely at one 
point. Clearly, in order to complete the proof, it suffices to prove that 
4>{G') = {/}, where G' is the commutator subgroup of G. There are 
three possibilities for the Jordan form of La- 

(i) . Suppose that La has two distinct eigenvalues Ai and A2, with 
corresponding eigenvectors Vi and V2; Laivi) = XiVi. With respect to 
the basis {vi,V2}, the matrix La is diagonal. Since and L^. preserve 
each eigenspace of La, they are diagonal too. In particular, they com- 
mute. Now from the braid relation LbLcLb = LcL^Lc, we get Lb = Lc, 
or (f){tbt~^) = L Since G' is generated normally by tbt~^ (c.f. Theo- 
rem [33]), we conclude that is trivial 

(ii) . If the matrix La has only one eigenvalue A and if the Jordan 
form of La is XI, then L^ = XI for each nonseparating simple closed 
curve X on S*. This is because L^ is conjugate to L^. Since the group 
G is generated by Dehn twists about such curves, we have that 4>{G) 
is cyclic. 

(iii) . Suppose finally that the Jordan form of La is not diagonal, so 
that the matrix of La is 

■ A 1 
A 

in some fixed basis. Because Lb and Lc preserve the eigenspace of 
La, with respect to the same basis, the matrices Lb and L^ are upper 
triangular whose diagonal entries are A. In particular, we have LbL^ = 
LcLb- From the braid relation LbL^Lb = L^LbLc again, we get Lb = L^, 
or (f){tbt^^) = I. From this we conclude that 4>{G') is trivial. 

This completes the proof of the proposition. □ 

Lemma 4.2. Let a,b,x,y be four nonseparating simple closed curves 
on S such that there is an orientation-preserving diffeomorphism f of 
S with f{x) = a and f{y) = b. Let X be an eigenvalue of La = (f){ta)- 
Then = El if and only if El = El. 

Proof. Let F = The assumptions f{x) = a and f{y) = b imply 

that ftxf^^ = ta and ftyf~^ = tb, and hence FL^F^^ = La and 
FLyF~^ = Lb- Therefore, 

= E^{La) = E^{FL,F-') = F{El) 

and 

E\ = E.iLb) = E.iFLyF'') = F{El). 
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The lemma now follows from these two. □ 

Proposition 4.3. If g = 2 then the image of any homomorphism 

(j) : Mod(S') — )■ GL(3, C) is a quotient of the cyclic group Ziq 

Proof. Let G = Mod(S'), and let G' denote the commutator subgroup 
of G. Since Hi{G, Z) is isomorphic to Ziq, it suffices to prove that 4>{G) 
is abelian, or equivalently that (j){G') is trivial. 

Since is generated normally by a single element txt~^ for any 

two nonseparating simple closed curves x and y intersecting at one 
point, it is also sufficient to find two such curves with (i>{t.J.y^) = I. 

Let a be a nonseparating simple closed curve on S. The Jordan form 
of La is one of the following six matrices: 



/ Ai 
(i) A2 
V A3 





('0 
















) • (") 1 




A 





1 . (iii) 1 









[0 





Ay 










/A10\ /A00\ /A00\ 

(iv) A 1 , (v) A 1 , (vi) A . 

\ooAy \ooAy yoo/^y 

Here, distinct notations represent distinct eigenvalues. In each case we 
fix a basis with respect to which the matrix La is in its Jordan form. 
Recall that the eigenspace of L^ corresponding to an eigenvalue A is 
denoted by E^. We now analyze each case. 

(i) . In this case, each eigenspace of La is 1-dimensional. Let bi 
and 62 be two nonseparating simple closed curves intersecting at one 
point such that each bi is disjoint from a. Then each eigenspace is 
invariant under each L^^, so that L^^ are diagonal. In particular, and 
Lfcj commute. Now the braid relation Liy^Li,.^Li,-^ = Li,.^Li)^Li,^ implies 
that Lfej = Lfej. Thus, we have 0(I/bJ = ^(LbJ, i.e. (pitb^tl^) = L 

(ii) . If 6 is a nonseparating simple closed curve on then is 
conjugate to La, so that L;, = XL Since G is generated by all such 
Dehn twists, we get that (j){G) is cyclic. 

(iii) . Let 61 and 62 be two nonseparating simple closed curves inter- 
secting at one point such that they are disjoint from a. The matrices 

and L^^ preserve each eigenspace and E"^, so that they are of 
the form 

(Xi Wi\ 
yi Ui \ . 
Zi J 

The braid relation Lb-^Lb^Lf,-^ = Lb^L^^Lb^ implies that x = xi = X2, 
y = yi = Z/2 and z ^ zi ^ Z2. The equality L^.L^ = L^L;,. then gives 
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Wi = and y = z. Hence, we have x = X, y = z = /j,, and so 

/ A \ 

Lb^= I Ui . 

\0 fi J 

But then we have Lh^^Lb^ = Lb^Lh-^. The braid relation Lb^Lb^Lb^ = 
Lb^Lb^Lb^ again imphes that Lb^ = Lb^-, and hence 4>{tb^tl^) = I- 

(iv) . Let hi and 62 be two nonseparating simple closed curves inter- 
secting at one point such that they are disjoint from a. In this case, 
ker(La — A/) = E'^ is 1-dimensional, ker(La — AJ)^ is 2-dimensional, 
and they are L^. -invariant for i = 1,2. It follows that 

(A * * \ 
OA*. 
A / 

Since there is the braid relation Lb^Lb^Lb^ = Lb^Lb^Lb^, Lemma [2711 
implies that Lb^ = Lb2- Hence, (t){tb^t~^^) = I- 

(v) . The eigenspace E'^ is 2-dimensional in this case. Suppose first 
that E^ ^ El for some (hence all) nonseparating simple closed curve 
h intersecting a at one point. Choose two nonseparating simple closed 
curves Ci and C2 disjoint from a U 6 such that Ci intersects C2 at one 
point. Then E'^ fl E\ and -E^ are L^. -invariant subspaces, so that with 
respect to a suitable basis 

(A * * \ 
OA*. 
A / 

Note that we have the braid relation Lc^Lc^L^^ = Lc2Lc^Lc2- We now 
use Lemma Y2A\ to conclude that L^,^ = Lc^, so that 4>(tcit~^) = I- 

If E'^ = E\ for some (hence all) h intersecting a at one point, then 
E^ = Ef for all nonseparating simple closed curves x. This is because, 
by Theorem 13.11 there is a sequence a = oq, ai, 02, . . . , = x of nonsep- 
arating simple closed curves such that aj_i intersects at one point 
for all 1 < i < k, and E'"*"^ = Since G is generated by Dehn 

twists about nonseparating simple closed curves, E^ is (/)(G)-invariant, 
so that induces a homomorphism (f) : G ^ GL(i?^) = GL(2,C). By 
Proposition 14. 0(G) is cyclic, and hence 0(/) = / all / G G'. Thus 
the matrix of 0(/) is of the form 

1 zi \ 

1 Z2 . 
Z3 / 
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Since the subgroup of GL(3, C) consisting of upper triangular matrices 
are solvable and since G' is perfect, (f){G') is trivial. 

(vi). In this last case, the eigenspace E'^ is, again, 2-dimensional. If 
E'^ = E\ for some nonseparating simple closed curve h intersecting a 
at one point, then from Theorem 13.11 and Lemma 14.21 we obtain that 
i?^ = E^ for all nonseparating simple closed curves x. We conclude 
now as in the case (v) that (piC) is trivial. 

Suppose finally that E'^ ^ E\ for some (hence all) nonseparating 
simple closed curve h intersecting a at one point. By Lemma 14. 2[ 
E^ 7^ E\ for all nonseparating simple closed curves x and y intersecting 
once. Let a = C4 and b = C5. Choose three nonseparating simple closed 
curves ci, C2, C3 such that 

• Ci intersects Cj at one point if |z — j| = 1, and 

• Cj is disjoint from cj if \i — j\ > 2. 

Let vi E E^n E\, t>2 G E^ and V3 G E'^ so that {vi,V2,V3} is a basis. 
With respect to this basis, the matrix of La is its Jordan matrix. Since 
E'^ri E'^, E'^ and i?^ are Lc,-invariant for z = 1, 2, we have 



gives us Xi = X2 = x,yi = 1/2 = y and zi = Z2 = z. 

If z = fi then X = y = X, and hence Lc^Lc2 = -Lc2-^ci- Now the braid 
relation ([T]) implies again that = Lc^- It follows that (piG') is trivial 
as above. 

Suppose that z = X. We will show that this case is not possible by 
arriving at a contradiction. If x = A then y = fi. But then we have 
E^^ = E'^, which is a contradiction. If x = /i then y = X. Since E"^^ is 
Lc3-invariant, we have 



Now the braid relation LaLc^La = Lc^LaLc^ gives u = X, while the 
braid relation L^^L^^Lc^ = Lc^Lc^Lc^ gives u = fi. Since A 7^ /x, we get 
a contradiction again. 





This completes the proof of the proposition. 



□ 



Finally, we are in a position to prove Theorem [TJ 
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Proof of Theorem^ li g = I then n = 1, and : Mod{S) GL(1, C). 
Since GL(1,C) is abelian, factors through i7i(Mod(5'); Z). The the- 
orem now follows from Theorem 13.61 If g = 2 then n < 3. These 
cases are proved in Propositions 14.11 and 14.31 We assume that g > 3 
and that the theorem holds true for all surfaces of genus g — 1- Since 
GL{k — 1, C) is isomorphic to a subgroup of GL{k,C), it suffices to 
prove the theorem for n = 2g — 1. 

In what follows R denotes a subsurface of S diffeomorphic to a com- 
pact connected surface of genus g — 1 with one boundary component. 
We embed Mod(i?) into Mod(S') by extending self-diffeomorphisms of 

to 5 by the identity. We set G = Mod(^) and Hr = Mod(i?). 

If, for some subsurface R, there exists a 0(if/j)-invariant subspace V 
of dimension r with 2 < r < n — 2, then induces homomorphisms 0i : 
Hr GL{V) = GL(r, C) and ^2 ■ Hr ^ GL(C"/y) = GL(n - r, C). 
Note that r < 2{g — 1) — 1 and n — r < 2{g — 1) — 1. By assumption, 
the image of each 0j is cyclic. In particular, if b and c are two simple 
closed curves on R intersecting at one point, then 0j(tfe) = (piifc) by 
Lemma 1X71 That is, 4>i(tht~^) = I. Since the commutator subgroup 
H'^ of Hr is generated normally by tf,t~^ we get that 0j(/) = / for all 
/ G H'^. It follows that, with respect to some basis of C", 



for all / G H'j^. Since the subgroup of GL(n, C) consisting of such 
matrices is abelian and since Hr is perfect, we conclude that (^{H'r) is 
trivial. In particular, we have (j){tbt~^) = I. Since G' = G is generated 
normally by tbt~^, 4>{G) is trivial. 

We now fix a subsurface R of genus g — 1 with one boundary com- 
ponent. Let a and b be two nonseparating simple closed curves on S 
intersecting at one point such that a U 6 is disjoint from R. 

CASE 1. Suppose that there is a subspace V of dimension r with 
2 < r < 2 which is a direct sum of eigenspaces of La (Note that there 
exists such a subspace if La has at least three distinct eigenvalues.). 
Then V is 0(if/j) -invariant. Hence, (f){G) is trivial. 

CASE 2. Suppose that there is no subspace V as in CASE 1. In 
particular. La has at most two eigenvalues and each eigenspace of La is 
either 1-dimensional, or (n — l)-dimensional, or n-dimensional. The 
Jordan form of La is one of the following four matrices: 
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/ A 1 
A 1 



\ 





/AO 
A 



\ 




(i) A/„, (ii) 



111 





V 



A 
A 1 
A / 




V 



A 1 
OA/ 



IV 



( 



XL 







n-l 



:) 



We fix a basis so tliat the matrix La is equal to its Jordan form. 

In tfie case (i), if a; is a nonseparating simple closed curve on S, then 
Lj. = XL since it is conjugate to L^. Since G is generated by Dehn 
twists about nonseparating simple closed curves, 0(G) is cyclic, and 
hence it is trivial. 

In the case (ii), the subspace ker(La — AJ)^ is a 0(ifij) -invariant 
subspace of dimension 2, so that 0(G) is trivial. 

It remain to consider the cases (iii) and (iv). In these cases, the 
eigenspace E'^ is of dimension n — 1. The eigenspace E'^ is also of 
dimension n-l. If ^ then E^ f] E\ is a 0(if^)-invariant 
subspace of dimension n — 2(> 1). Hence, 0(G) is trivial. 

Suppose finally that E'^ = E\. It follows from Lemma 14.21 that 
E^ = Ef for any two nonseparating simple closed curves x and y on 
S intersecting at one point. We then conclude from Theorem 13. II that 
E'^ = Ef for all such nonseparating x on S. Since Mod(5') is generated 
by Dehn twists about nonseparating simple closed curves, it follows 
that for each / e G, the matrix 0(/) is upper triangular. Hence, 0(G) is 
contained in the subgroup consisting of upper triangular matrices. But 
this subgroup of GL(n, C) is solvable. Since G is perfect, we conclude 
from Lemma [2.31 that 0(G) is trivial. 

This completes the proof of Theorem [TJ □ 



The purpose of this section is to prove Theorem HI So let : 
Mod(A^) — )■ GL(?T,, C) be a homomorphism, where is a nonorientable 
surface of genus g > 3 with p > marked points, and n < g — 2 for 
odd g and n < g — 3 for even g. 

Proof of Theorem^ li g = 3 oi g = 4: then n = 1, and GL(1, C) is 
abelian, so that factors through the first homology iJi(Mod(A^); Z). 
Since i/i(Mod(A^); Z) is finite (c.f. [El [H] ) , the result follows. So we 
assume that g > 5. 



5. Nonorientable surfaces 
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Let T denote the subgroup of the mapping class group Mod(A^). 
Write (yf = 2r + 1 if (7 is odd and g = 2r + 2 if g is even. Hence, we have 
r > 2 and n < 2r — 1. 

Let S" be a compact orientable surface of genus r with one bound- 
ary component. Embed S into A^, so that the boundary of S bounds 
a Mobius band (resp. Klein bottle with one boundary) on with p 
marked points if g is odd (resp. even). Extending diffeomorphisms 
5* to iV by the identity induces a homomorphism rj : Mod(S') — 
Mod(A^). Then the composition (pi] is a homomorphism from Mod(S') 
to GL(n,C). 



If r > 3 ((? > 7) then (prj is trivial by Theorem [T] It follows that 
4>{ta) = I for any Dehn twist supported on S. If 6 is a two-sided nonsep- 
arating simple closed curve on N whose complement in nonorientable, 
it follows from Theorems 3.1 and 5.3 in [T^ that a Dehn twist th about 
b is conjugate to a Dehn twist supported on S. Hence, 0(tb) is trivial 
for all such b. Since T is generated by such Dehn twists (c.f. [H], proof 
of Theorem 5.12), we get that 0(T) is trivial. We also know that the 
index of T in Mod(A^) is p\ ■ 2^"*"^ ([T^? Corollary 6.2). The conclusion 
of the theorem now follows. 

If r = 2 (^f = 5 or (7 = 6) then (prj is cyclic by Theorem [H It follows 
that <t>{tat~^^) = I for any two nonseparating simple closed curves on 
5*. Let X and y be two two-sided nonseparating simple closed curves 
on intersecting at one point such that the complement of each is 
nonorientable. It can easily be shown that there is a diffeomorphism / 
of such that f{x Uy) C S, so that t^t'^ can be conjugated to tati^^, 
where a and b are on S. Hence, (pit^ty^) = I, or 4>(tx) = (pity). We 
now apply Theorem 3.1 in [14] to conclude that 0(tx) = 0('^j/) for ciU 
two-sided nonseparating simple closed curves whose complements are 
nonorientable. (Such simple closed curves are called essential in [Hj.) 
Since T is generated by such Dehn twists, we get that 0(T) is cyclic, 
so that 0(T') = {/}, where T' is the commutator subgroup. Stukow 
proved that the index of T' in T is 2 (c.f. [20j, Theorem 8.1). We 
conclude that 0(Mod(A^)) is a finite group of order at most p! ■ 2^^"^. 

This finishes the proof of Theorem m □ 

Remark 5.1. \i g >1 and if A^ is closed, then the above proof shows 
that the image of in Theorem H] is either trivial or is isomorphic to Z2. 



Mod(Ar) 




7] 



Mod(5) 
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In fact, it is easy to find a homomorphism whose image is Z2; send all 
Dehn twists to the identity and crosscap slides (=y-homeomorphisms) 
to an element of order 2. 

6. HOMOMORPHISMS TO Aut(F„) AND Out(F„) 

We prove Theorem [5] in this section. The proof in the case n = 1 is 
trivial. So we assume that 2 < n < 2g — 1. 

Let Fn denote the free group of rank n. The action of the automor- 
phism group Aut(F„) of Fn on the abelianization of F„ gives rise to a 
surjective homomorphism rj : Aut(F„) — )■ GL(n,Z). The kernel of this 
map is usually denoted by I An. Let Out(F„) denote the group of outer 
automorphisms of F„, so that it is the quotient of Aut(-F„) with the 
(normal) subgroup lnn(F„) of inner automorphisms. 

Suppose that 5 is a closed surface of genus 2. Choose five pairwise 
nonisotopic nonseparating simple closed curves ci, C2, C3, C4, C5 on S such 
that Ci is disjoint from Cj if |i — j| > 2 and that q intersects Cj at 
one point if \i — j\ = 1. Let ti denote the right Dehn twists about 
Cj. Then the group Mod(S') is generated by ^1,^2,^3,^4,^5- Let us set 
a = tit2^3^4^5- It is well-known that cr is a torsion element of order six. 
One can easily show that 

(2) atiO'^ = U+i 

for each 1 < i < 4. 

Lemma 6.1. Let g = 2. The normal closure of in Mod(S') is equal 
to the commutator subgroup ofMod{S). 

Proof. Let denote the the normal closure of o"^ in Mod(S'), the in- 
tersection of all normal subgroups containing o"^. Since any right Dehn 
twist in Mod(S') about a nonseparating simple closed curve maps to 
the generator of ifi(Mod(S'), Z) under the natural homomorphism and 
since is a product of 10 such Dehn twists, we see that o"^ is contained 
in [Mod(5),Mod(5)]. Hence, A^ c [Mod(^), Mod(5)]. 

Let q : Mod(S') — )■ Mod(5')/A^ denote the quotient map. The equal- 
ity ([2]) implies that = qit^) = q{t^) and 5(^2) = qit^)- On the 
other hand, from the braid relation we get 

q{ti)q{U)q{ti) = qih)q{t2)q{t,) 

= qitihh) 

= qihhh) 

= q{t2)q{ti)q{t2) 

= q{ti)q{ti)q{ti). 
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Since ti and ^4 commute, we obtain = qit^). It follows that 

Mod(5')/A^ is cyclic. In particular, N contains [Mod(5'), Mod(S')], fin- 
ishing the proof of the lemma. □ 

Proof of Theorem\^ Suppose that H = Aut(F„). Let be the com- 
position of if with 1], so that we have a commutative diagram: 

Mod{S) 




1 >IaJ^ ^Aut(F„)^^GL(n,Z) ^ 1. 

We now apply Theorem [H If g > 3 then is trivial, implying that 
the image of ip is contained in I An- Since I An is torsion-free by a result 
of Baumslag- Taylor [1], all torsion elements in Mod(S') are contained 
in the kernel of (p. Since Mod(S') is is generated by torsion elements 
(cf. [HI [ini US]), we conclude that (p is trivial. 

Suppose now that g = 2. Theorem [1] gives us that the image of 
(f) is cyclic, implying that the commutator subgroup of Mod(S') is 
contained in the kernel of (p. Therefore, (fiN) is contained in I An. 
Since I An is torsion-free and since G A^ is a torsion element (of 
order 3), we have </3(ct^) = 1. It follows from Lemma 16.1 1 that (p{N) 
is trivial. Consequently, factors through the natural homomorphism 
Mod(5') — 7- Hi(Mod{S) , Z), and so the image of <^ is a quotient of Ziq. 

This completes the proof of Theorem [5] for Aut(F„). 

The case H = Out(F„) is completely similar and uses the fact that 
the subgroup IAn/Inia{Fn) is torsion-free (cf. [1]). □ 
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